Skip to content

[WIP][GLUTEN-8055][VL] Support read Iceberg equality delete file MOR table#8056

Draft
liujiayi771 wants to merge 1 commit intoapache:mainfrom
liujiayi771:iceberg-equality-delete-mor
Draft

[WIP][GLUTEN-8055][VL] Support read Iceberg equality delete file MOR table#8056
liujiayi771 wants to merge 1 commit intoapache:mainfrom
liujiayi771:iceberg-equality-delete-mor

Conversation

@liujiayi771
Copy link
Contributor

@liujiayi771 liujiayi771 commented Nov 26, 2024

What changes were proposed in this pull request?

Fix #8055. Support reading Iceberg equality delete file MOR table.

How was this patch tested?

Iceberg equality delete file MOR tablescan only be generated through Flink, and the Iceberg table metadata stores absolute paths. Therefore, tables written by Flink cannot create corresponding external table in unit tests for testing. We need to think of some methods to test the reading of Iceberg equality delete file MOR table.

@github-actions github-actions bot added CORE works for Gluten Core VELOX DATA_LAKE labels Nov 26, 2024
@github-actions
Copy link

github-actions bot commented Nov 26, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link

github-actions bot commented Nov 26, 2024

Run Gluten Clickhouse CI on x86

@liujiayi771
Copy link
Contributor Author

liujiayi771 commented Nov 26, 2024 •
edited
Loading

@FelixYBW @rui-mo I have tested that facebookincubator/velox#11088 can work with Gluten. The current PR can be reviewed after the related PR in Velox upstream is merged.

@liujiayi771 liujiayi771 changed the title [WIP][VL] Support read Iceberg equality delete file MOR table [WIP][GLUTEN-8055][VL] Support read Iceberg equality delete file MOR table Nov 26, 2024
@liujiayi771 liujiayi771 force-pushed the iceberg-equality-delete-mor branch from d3a415b to a76cc64 Compare November 26, 2024 13:29
@github-actions
Copy link

github-actions bot commented Nov 26, 2024

Run Gluten Clickhouse CI on x86

@liujiayi771 liujiayi771 force-pushed the iceberg-equality-delete-mor branch from a76cc64 to 8d05523 Compare November 27, 2024 02:32
@github-actions
Copy link

github-actions bot commented Nov 27, 2024

Run Gluten Clickhouse CI on x86

@liujiayi771
Copy link
Contributor Author

liujiayi771 commented Nov 27, 2024 •
edited
Loading

lowerBounds and upperBounds in IcebergDeleteFile are not currently used in Velox, so I have removed the related code for now. I will add it back later when Velox supports these two variables, but I have kept their definitions in the protobuf.

@zhouyuan zhouyuan mentioned this pull request Jan 8, 2025
Open
25 tasks
@github-actions
Copy link

github-actions bot commented Jan 12, 2025

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Jan 12, 2025
@liujiayi771 liujiayi771 force-pushed the iceberg-equality-delete-mor branch from 8d05523 to 6ebaa82 Compare January 13, 2025 02:41
@github-actions
Copy link

github-actions bot commented Jan 13, 2025

Run Gluten Clickhouse CI on x86

@liujiayi771 liujiayi771 removed the stale stale label Jan 13, 2025
@liujiayi771 liujiayi771 force-pushed the iceberg-equality-delete-mor branch from 6ebaa82 to 2e26b4d Compare January 13, 2025 03:17
@github-actions
Copy link

github-actions bot commented Jan 13, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link

github-actions bot commented Feb 28, 2025

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Feb 28, 2025
@github-actions
Copy link

github-actions bot commented Mar 11, 2025

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

@github-actions github-actions bot closed this Mar 11, 2025
@zhouyuan zhouyuan reopened this Apr 2, 2025
@zhouyuan zhouyuan removed the stale stale label Apr 2, 2025
@github-actions
Copy link

github-actions bot commented Apr 2, 2025

Run Gluten Clickhouse CI on x86

@zhouyuan zhouyuan mentioned this pull request Apr 4, 2025
Open
@github-actions
Copy link

github-actions bot commented May 18, 2025

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label May 18, 2025
@github-actions github-actions bot added the stale stale label Jul 14, 2025
@zhouyuan zhouyuan removed the stale stale label Jul 14, 2025
@zhouyuan
Copy link
Member

zhouyuan commented Aug 8, 2025

Note: the equality deletion vector support patch is available in IBM/Velox now

@github-actions
Copy link

github-actions bot commented Sep 23, 2025

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Sep 23, 2025
@github-actions
Copy link

github-actions bot commented Oct 4, 2025

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

@github-actions github-actions bot closed this Oct 4, 2025
@zhouyuan zhouyuan reopened this Oct 6, 2025
@github-actions
Copy link

github-actions bot commented Oct 6, 2025

Run Gluten Clickhouse CI on x86

@github-actions github-actions bot removed the stale stale label Oct 7, 2025
@github-actions
Copy link

github-actions bot commented Nov 21, 2025

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Nov 21, 2025
@zhouyuan zhouyuan removed the stale stale label Nov 25, 2025
@zhouyuan
Copy link
Member

zhouyuan commented Dec 9, 2025

Hi @liujiayi771 The equality delete patch is porte to IBM/Velox as the enhanced feature, so we may make some progress by adding in the scope of ENABLE_EHNANCED_FEATURE. Would you like to rebase and add this compile scope? I could help on this if required.

@liujiayi771
Copy link
Contributor Author

liujiayi771 commented Dec 10, 2025

@zhouyuan OK, I'll rebase the code today.

@liujiayi771 liujiayi771 force-pushed the iceberg-equality-delete-mor branch from 2e26b4d to 45ba968 Compare December 10, 2025 08:40
@github-actions github-actions bot added BUILD and removed CORE works for Gluten Core labels Dec 10, 2025
@github-actions
Copy link

github-actions bot commented Dec 10, 2025

Run Gluten Clickhouse CI on x86

@liujiayi771
Copy link
Contributor Author

liujiayi771 commented Dec 10, 2025

@zhouyuan We can move Velox(ClickHouse)IcebergMOREqualityDeletionSuite to gluten-iceberg after 11088 is merged into velox upstream.

@liujiayi771 liujiayi771 force-pushed the iceberg-equality-delete-mor branch from 45ba968 to 0d442bf Compare December 10, 2025 08:43
@github-actions
Copy link

github-actions bot commented Dec 10, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link

github-actions bot commented Jan 25, 2026

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Jan 25, 2026
@github-actions
Copy link

github-actions bot commented Feb 4, 2026

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

@github-actions github-actions bot closed this Feb 4, 2026
@zhouyuan zhouyuan reopened this Feb 6, 2026
@zhouyuan zhouyuan removed the stale stale label Feb 6, 2026
@github-actions
Copy link

github-actions bot commented Feb 6, 2026

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

BUILD DATA_LAKE VELOX

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[VL] Support read Iceberg equality delete file MOR table

2 participants

@liujiayi771 @zhouyuan